{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Fraud Detection Systems\n",
    "\n",
    "Fraud detection is one of the top priorities for banks and financial institutions, which can be addressed using machine learning. According to [a report published by Nilson](https://nilsonreport.com/upload/content_promo/The_Nilson_Report_Issue_1118.pdf), in 2017 the worldwide losses in card fraud related cases reached 22.8 billion dollars. The problem is forecasted to get worse in the following years, by 2021, the card fraud bill is expected to be 32.96 billion dollars.\n",
    "\n",
    "In this tutorial, we will use the [credit card fraud detection dataset](https://www.kaggle.com/mlg-ulb/creditcardfraud) from Kaggle, to identify fraud cases. We will use a [gradient boosted tree](https://blogs.technet.microsoft.com/machinelearning/2017/07/25/lessons-learned-benchmarking-fast-machine-learning-algorithms/) as a machine learning algorithm. And finally, we will create a simple API to operationalize (o16n) the model.\n",
    "\n",
    "We will use the gradient boosting library [LightGBM](https://github.com/Microsoft/LightGBM), which has recently became one of the most popular libraries for top participants in [Kaggle competitions](https://github.com/Microsoft/LightGBM/tree/a39c848e6456d473d2043dff3f5159945a36b567/examples). \n",
    "\n",
    "Fraud detection problems are known for being extremely imbalanced. <a href=\"https://en.wikipedia.org/wiki/Boosting_(machine_learning)\">Boosting</a> is one technique that usually works well with these kind of datasets. It iteratively creates weak classifiers (decision trees) weighting the instances to increase the performance. In the first subset, a weak classifier is trained and tested on all the training data, those instances that have bad performance are weighted to appear more in the next data subset. Finally, all the classifiers are ensembled with a weighted average of their estimates.\n",
    "\n",
    "In LightGBM, there is a [parameter](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst#objective-parameters) called `is_unbalanced` that automatically helps you to control this issue. \n",
    "\n",
    "LigtGBM can be used with or without [GPU](https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html). For small datasets, like the one we are using here, it is faster to use CPU, due to IO overhead. However, I wanted to showcase the GPU alternative, which is trickier to install, in case anyone wants to experiment with bigger datasets.\n",
    "\n",
    "To install the dependencies in Linux:\n",
    "```bash\n",
    "$ sudo apt-get update\n",
    "$ sudo apt-get install cmake build-essential libboost-all-dev -y\n",
    "$ conda env create -n fraud -f conda.yaml\n",
    "$ source activate fraud\n",
    "(fraud)$ python -m ipykernel install --user --name fraud --display-name \"Python (fraud)\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 13:19:00) \n",
      "[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]\n",
      "Numpy version: 1.13.3\n",
      "Pandas version: 0.22.0\n",
      "LightGBM version: 2.1.1\n",
      "Sklearn version: 0.19.1\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "import json\n",
    "import pandas as pd\n",
    "from collections import Counter\n",
    "import requests\n",
    "from IPython.core.display import display, HTML\n",
    "import lightgbm as lgb\n",
    "import sklearn\n",
    "import aiohttp\n",
    "import asyncio\n",
    "from utils import (split_train_test, classification_metrics_binary, classification_metrics_binary_prob,\n",
    "                   binarize_prediction, plot_confusion_matrix, run_load_test, read_from_sqlite)\n",
    "from utils import BASELINE_MODEL, PORT, TABLE_FRAUD, TABLE_LOCATIONS, DATABASE_FILE\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Numpy version: {}\".format(np.__version__))\n",
    "print(\"Pandas version: {}\".format(pd.__version__))\n",
    "print(\"LightGBM version: {}\".format(lgb.__version__))\n",
    "print(\"Sklearn version: {}\".format(sklearn.__version__))\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "The first step is to load the dataset and analyze it.\n",
    "\n",
    "For it, before continuing, **you have to run the notebook [data_prep.ipynb](data_prep.ipynb)**, which will generate the SQLite database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = 'SELECT * FROM ' + TABLE_FRAUD\n",
    "df = read_from_sqlite(DATABASE_FILE, query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (284807, 31)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.359807</td>\n",
       "      <td>-0.072781</td>\n",
       "      <td>2.536347</td>\n",
       "      <td>1.378155</td>\n",
       "      <td>-0.338321</td>\n",
       "      <td>0.462388</td>\n",
       "      <td>0.239599</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.363787</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.018307</td>\n",
       "      <td>0.277838</td>\n",
       "      <td>-0.110474</td>\n",
       "      <td>0.066928</td>\n",
       "      <td>0.128539</td>\n",
       "      <td>-0.189115</td>\n",
       "      <td>0.133558</td>\n",
       "      <td>-0.021053</td>\n",
       "      <td>149.62</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.191857</td>\n",
       "      <td>0.266151</td>\n",
       "      <td>0.166480</td>\n",
       "      <td>0.448154</td>\n",
       "      <td>0.060018</td>\n",
       "      <td>-0.082361</td>\n",
       "      <td>-0.078803</td>\n",
       "      <td>0.085102</td>\n",
       "      <td>-0.255425</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.225775</td>\n",
       "      <td>-0.638672</td>\n",
       "      <td>0.101288</td>\n",
       "      <td>-0.339846</td>\n",
       "      <td>0.167170</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>-0.008983</td>\n",
       "      <td>0.014724</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.358354</td>\n",
       "      <td>-1.340163</td>\n",
       "      <td>1.773209</td>\n",
       "      <td>0.379780</td>\n",
       "      <td>-0.503198</td>\n",
       "      <td>1.800499</td>\n",
       "      <td>0.791461</td>\n",
       "      <td>0.247676</td>\n",
       "      <td>-1.514654</td>\n",
       "      <td>...</td>\n",
       "      <td>0.247998</td>\n",
       "      <td>0.771679</td>\n",
       "      <td>0.909412</td>\n",
       "      <td>-0.689281</td>\n",
       "      <td>-0.327642</td>\n",
       "      <td>-0.139097</td>\n",
       "      <td>-0.055353</td>\n",
       "      <td>-0.059752</td>\n",
       "      <td>378.66</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.966272</td>\n",
       "      <td>-0.185226</td>\n",
       "      <td>1.792993</td>\n",
       "      <td>-0.863291</td>\n",
       "      <td>-0.010309</td>\n",
       "      <td>1.247203</td>\n",
       "      <td>0.237609</td>\n",
       "      <td>0.377436</td>\n",
       "      <td>-1.387024</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.108300</td>\n",
       "      <td>0.005274</td>\n",
       "      <td>-0.190321</td>\n",
       "      <td>-1.175575</td>\n",
       "      <td>0.647376</td>\n",
       "      <td>-0.221929</td>\n",
       "      <td>0.062723</td>\n",
       "      <td>0.061458</td>\n",
       "      <td>123.50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-1.158233</td>\n",
       "      <td>0.877737</td>\n",
       "      <td>1.548718</td>\n",
       "      <td>0.403034</td>\n",
       "      <td>-0.407193</td>\n",
       "      <td>0.095921</td>\n",
       "      <td>0.592941</td>\n",
       "      <td>-0.270533</td>\n",
       "      <td>0.817739</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.009431</td>\n",
       "      <td>0.798278</td>\n",
       "      <td>-0.137458</td>\n",
       "      <td>0.141267</td>\n",
       "      <td>-0.206010</td>\n",
       "      <td>0.502292</td>\n",
       "      <td>0.219422</td>\n",
       "      <td>0.215153</td>\n",
       "      <td>69.99</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Time        V1        V2        V3        V4        V5        V6        V7  \\\n",
       "0   0.0 -1.359807 -0.072781  2.536347  1.378155 -0.338321  0.462388  0.239599   \n",
       "1   0.0  1.191857  0.266151  0.166480  0.448154  0.060018 -0.082361 -0.078803   \n",
       "2   1.0 -1.358354 -1.340163  1.773209  0.379780 -0.503198  1.800499  0.791461   \n",
       "3   1.0 -0.966272 -0.185226  1.792993 -0.863291 -0.010309  1.247203  0.237609   \n",
       "4   2.0 -1.158233  0.877737  1.548718  0.403034 -0.407193  0.095921  0.592941   \n",
       "\n",
       "         V8        V9  ...         V21       V22       V23       V24  \\\n",
       "0  0.098698  0.363787  ...   -0.018307  0.277838 -0.110474  0.066928   \n",
       "1  0.085102 -0.255425  ...   -0.225775 -0.638672  0.101288 -0.339846   \n",
       "2  0.247676 -1.514654  ...    0.247998  0.771679  0.909412 -0.689281   \n",
       "3  0.377436 -1.387024  ...   -0.108300  0.005274 -0.190321 -1.175575   \n",
       "4 -0.270533  0.817739  ...   -0.009431  0.798278 -0.137458  0.141267   \n",
       "\n",
       "        V25       V26       V27       V28  Amount  Class  \n",
       "0  0.128539 -0.189115  0.133558 -0.021053  149.62      0  \n",
       "1  0.167170  0.125895 -0.008983  0.014724    2.69      0  \n",
       "2 -0.327642 -0.139097 -0.055353 -0.059752  378.66      0  \n",
       "3  0.647376 -0.221929  0.062723  0.061458  123.50      0  \n",
       "4 -0.206010  0.502292  0.219422  0.215153   69.99      0  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Shape: {}\".format(df.shape))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the dataset is extremely imbalanced. The minority class counts for around 0.002% of the examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    284315\n",
       "1       492\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.998273\n",
       "1    0.001727\n",
       "Name: Class, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to split the dataset into train and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(227845, 30)\n",
      "(56962, 30)\n",
      "(227845,)\n",
      "(56962,)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = split_train_test(df.drop('Class', axis=1), df['Class'], test_size=0.2)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    227451\n",
      "1       394\n",
      "Name: Class, dtype: int64\n",
      "0    0.998271\n",
      "1    0.001729\n",
      "Name: Class, dtype: float64\n",
      "0    56864\n",
      "1       98\n",
      "Name: Class, dtype: int64\n",
      "0    0.99828\n",
      "1    0.00172\n",
      "Name: Class, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(y_train.value_counts())\n",
    "print(y_train.value_counts(normalize=True))\n",
    "print(y_test.value_counts())\n",
    "print(y_test.value_counts(normalize=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with LightGBM - Baseline \n",
    "\n",
    "For this task we use a simple set of parameters to train the model. We just want to create a baseline model, so we are not performing here cross validation or parameter tunning. \n",
    "\n",
    "The details of the different parameters of LightGBM can be found in the [documentation](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst). Also, the authors provide [some advices](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters-Tuning.rst) on how to tune the parameters and prevent overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "lgb_train = lgb.Dataset(X_train, y_train, free_raw_data=False)\n",
    "lgb_test = lgb.Dataset(X_test, y_test, reference=lgb_train, free_raw_data=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {'num_leaves': 2**8,\n",
    "              'learning_rate': 0.1,\n",
    "              'is_unbalance': True,\n",
    "              'min_split_gain': 0.1,\n",
    "              'min_child_weight': 1,\n",
    "              'reg_lambda': 1,\n",
    "              'subsample': 1,\n",
    "              'objective':'binary',\n",
    "              #'device': 'gpu', # comment this line if you are not using GPU\n",
    "              'task': 'train'\n",
    "              }\n",
    "num_rounds = 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 45.1 s, sys: 7.68 s, total: 52.8 s\n",
      "Wall time: 11.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.train(parameters, lgb_train, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have the trained model, we can obtain some metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_prob = clf.predict(X_test)\n",
    "y_pred = binarize_prediction(y_prob, threshold=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[55773,  1091],\n",
       "       [   11,    87]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics = classification_metrics_binary(y_test, y_pred)\n",
    "metrics2 = classification_metrics_binary_prob(y_test, y_prob)\n",
    "metrics.update(metrics2)\n",
    "cm = metrics['Confusion Matrix']\n",
    "metrics.pop('Confusion Matrix', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"AUC\": 0.9322482105532139,\n",
      "    \"Accuracy\": 0.980653769179453,\n",
      "    \"F1\": 0.13636363636363638,\n",
      "    \"Log loss\": 0.6375216445628125,\n",
      "    \"Precision\": 0.07385398981324279,\n",
      "    \"Recall\": 0.8877551020408163\n",
      "}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbsAAAFsCAYAAABVUheJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XeclNXZxvHfRRMUEBTsosYaNIpK\n7BpbFCtqbLFhiSaWmESNmqJY4hujqcaowd4iWKJgidhrLIAKYoliixAVELAhKni/f5wzOC5bBtjd\n2Zm5vn7ms/Ocp8w9O7j3nPOcoojAzMysmrUrdwBmZmYtzcnOzMyqnpOdmZlVPSc7MzOrek52ZmZW\n9ZzszMys6jnZmVUASV0k3S7pA0k3LcR1DpR0T3PGVi6StpT0n3LHYZVBHmdn1nwkHQCcAKwFfAQ8\nB5wTEY8t5HUPBn4MbBYRsxc60DZOUgCrR8SEcsdi1cE1O7NmIukE4M/A/wFLA32Ai4CBzXD5lYBX\naiHRlUJSh3LHYJXFyc6sGUhaHDgLODYi/hkRn0TEFxFxe0T8PB+ziKQ/S/pffvxZ0iJ539aSJko6\nUdJkSe9IOizvOxM4HdhP0seSjpB0hqTril5/ZUlRSAKSDpX0uqSPJL0h6cCi8seKzttM0qjcPDpK\n0mZF+x6SdLakx/N17pHUq4H3X4j/5KL495C0s6RXJE2T9Mui4zeS9ISkGfnYCyV1yvseyYeNze93\nv6LrnyLpXeDKQlk+Z9X8Ghvk7eUkTZG09UJ9sFY1nOzMmsemQGfg1kaO+RWwCdAPWA/YCPh10f5l\ngMWB5YEjgL9J6hkRg0m1xWER0TUiLm8sEEmLARcAO0VEN2AzUnNq3eOWAO7Mxy4J/BG4U9KSRYcd\nABwGLAV0Ak5q5KWXIf0Olicl50uBg4ANgS2B0yStko+dA/wM6EX63W0HHAMQEVvlY9bL73dY0fWX\nINVyjyp+4Yh4DTgFuE7SosCVwNUR8VAj8VoNcbIzax5LAlObaGY8EDgrIiZHxBTgTODgov1f5P1f\nRMRdwMfAmgsYz5fAOpK6RMQ7EfFCPcfsArwaEddGxOyIuAF4Gdit6JgrI+KViPgUuJGUqBvyBen+\n5BfAUFIi+0tEfJRf/0VSkicixkTEk/l13wT+DnynhPc0OCI+y/F8TURcCkwAngKWJX25MAOc7Mya\ny/tArybuJS0HvFW0/VYum3uNOslyJtB1fgOJiE+A/YAfAe9IulPSWiXEU4hp+aLtd+cjnvcjYk5+\nXkhG7xXt/7RwvqQ1JN0h6V1JH5JqrvU2kRaZEhGzmjjmUmAd4K8R8VkTx1oNcbIzax5PAJ8BezRy\nzP9ITXAFfXLZgvgEWLRoe5ninRExMiK+S6rhvExKAk3FU4hp0gLGND8uJsW1ekR0B34JqIlzGu06\nLqkrqYPQ5cAZuZnWDHCyM2sWEfEB6T7V33LHjEUldZS0k6Tz8mE3AL+W1Dt39DgduK6hazbhOWAr\nSX1y55hfFHZIWlrSwHzv7jNSc+iX9VzjLmANSQdI6iBpP6AvcMcCxjQ/ugEfAh/nWufRdfa/B3xj\nPq/5F2B0RPyAdC/ykoWO0qqGk51ZM4mIP5DG2P0amAK8DRwH3JYP+Q0wGhgHPA88k8sW5LXuBYbl\na43h6wmqXY7jf8A00r2wusmEiHgf2BU4kdQMezKwa0RMXZCY5tNJpM4vH5FqncPq7D8DuDr31ty3\nqYtJGggM4Kv3eQKwQaEXqpkHlZuZWdVzzc7MzKqek52ZmVU9JzszM6t6TnZmZlb1nOzMzKzqeeZw\na3Hq0CXUqVu5w7D50O+bfcodgs2nZ58ZMzUiei/o+e27rxQxe55Z2OoVn04ZGREDFvS1ysHJzlqc\nOnVjkTWbHCplbcgj/76g3CHYfOrWuX3dqd/mS8yexSJr7V/SsbOe/WtTU7u1OU52ZmaWJmtTUzO2\nVS4nOzMzS1S93Tic7MzMLHHNzszMqptcszMzsyonoF37ckfRYpzszMyMVLNzM6aZmVU7N2OamVnV\nc83OzMyqmzuomJlZtfOgcjMzq36CdtWbEqr3nZmZ2fxp55qdmZlVM+F7dmZmVgN8z87MzKqbe2Oa\nmVkt8HRhZmZW1VTd04VVb53VzMzmj9qV9ijlUtKbkp6X9Jyk0blsCUn3Sno1/+yZyyXpAkkTJI2T\ntEHRdQbl41+VNKiofMN8/Qn53EYztZOdmZklhdpdU4/SbRMR/SKif94+Fbg/IlYH7s/bADsBq+fH\nUcDFKRwtAQwGNgY2AgYXEmQ+5sii8wY0FoiTnZmZMbeDSjPV7BowELg6P78a2KOo/JpIngR6SFoW\n2BG4NyKmRcR04F5gQN7XPSKejIgArim6Vr2c7MzMLCm9ZtdL0uiix1H1XC2AeySNKdq/dES8k5+/\nCyydny8PvF107sRc1lj5xHrKG+QOKmZmlpJY6dOFTS1qmmzIFhExSdJSwL2SXi7eGREhKRYk1AXh\nmp2ZmSXNeM8uIibln5OBW0n33N7LTZDkn5Pz4ZOAFYtOXyGXNVa+Qj3lDXKyMzOzpJnu2UlaTFK3\nwnNgB2A8MAIo9KgcBAzPz0cAh+RemZsAH+TmzpHADpJ65o4pOwAj874PJW2Se2EeUnSterkZ08zM\nkuYbZ7c0cGseDdAB+EdE3C1pFHCjpCOAt4B98/F3ATsDE4CZwGEAETFN0tnAqHzcWRExLT8/BrgK\n6AL8Kz8a5GRnZma5ibJ5Gvsi4nVgvXrK3we2q6c8gGMbuNYVwBX1lI8G1ik1Jic7MzNLqngGFSc7\nMzNDQLt21duNw8nOzMzyenblDqLlONmZmRkgmphesqI52ZmZGYCTnZmZVT8nOzMzq24CtXOyMzOz\nKibfszMzs1rgZGdmZlXPyc7MzKqek52ZmVU3Dyo3M7NqJ+TpwszMrPq5GdPMzKpf9eY6JzszMyMN\nKnfNzszMqp2TnZmZVT0nOzMzq2pCnhvTrJq9fOeZfPTJZ8z58ktmz/mSLQ48j1/9cGcO32szpkz/\nGIDBF45g5GMvsv9O/fnpoO3nnvut1Zdj0+//jtcnTuW+K342t3z5pXow9K5R/Pz3t/CDvbfgh/tu\nxZwvv+STmZ9x7G9u4OXX323191mtjj7qCO7+15307r0UTz8zDoBp06Zx6EH789+33qLPSitx9fXD\n6NmzJ9OnT+eYHx7BG6+/TufOnbno75fRd+11GrxOTanye3aKiHLHYFWu3aJLxSJr7lvuMBr08p1n\nsvmB5/H+jE/mlv3qhzvzyczP+PO19zd43tqrLceNfzyStXc/c559j19/Mif/4RYef+Y1ui3WmY8+\nmQXALt/5FkftsyUDj7uo+d9IM5ry5AXlDqFkjz36CF27duWoIw6dm6R+/ctT6NlzCU78+Sn84fzf\nMWPGdM4+51x+9YuT6bpYV37x69P5z39e5sSf/Jg77r63wetUkm6d24+JiP4Len6npVaLpfb+fUnH\nTrp4z4V6rXKo3hGEZi1s3wEbctPIZ+YpX63PUiy1RDcef+Y1gLmJDmCxLp0I/AWzOW2x5Vb07LnE\n18ruvH0EBx50CAAHHnQId4wYDsDLL73IVltvA8Caa67Ff996k8nvvdfgdWqNpJIelcjJzmpeRHD7\nRcfx+PUnc/hem88t/9H+W/H0sF9wyeAD6dGtyzzn7b3DBtx49+h5yvcZsAE33/P1JPjDfbfihRGD\nOecne3DieTc3/5uwr5ky+T2WWXZZAJZeZhmmTE4J7VvfWo/bh98KwOhRT/Pf/77FpEkTyxZnm6MS\nHxWoTSY7Sb0lPSXpWUlbtsD1H5JUbxVc0s2SvtHcr9lILL+ss/3vFnytMySd1EzX6iTpEUkVf993\nu8P+xGYH/I49jruIH+63JZtvsCqX3vQofXc7g433P5d3p37IuSfs9bVzvr3OSsyc9QUvvvbOPNfb\nZ8cN50mCf7/xEdbe/Ux+/ZfhnPqDAS36fuzrimsjJ/z8FGbMmMFmG23A3y+6kPX6rU/79u3LHGHb\nIKXpwkp5VKK2GvV2wPMRsX5EPFq8Q1KL/cuUtDbQPiJeb6nXqMfXkl1EbNaKr73AIuJz4H5gv3LH\nsrD+N+UDAKZM/5gRD4zj22uvzORpH/Hll0FEcMU/H6f/Oit97Zz6EhrAt9ZYng7t2/PsS2/X+1o3\njhzDbluv2/xvwr6m91JL8+476YvIu++8Q6/eSwHQvXt3Lrn0Cv799DMMueJqpk6ZwsqrtNp32zbP\nzZjzSdLKkl6SdKmkFyTdI6lL3tdP0pOSxkm6VVLPOuf2A84DBkp6TlIXSR9L+oOkscCmkk6XNErS\neElDlH/7xTU2Sb0kvZmfd5E0NMd0KzBvm1RyIDC8KJaPJZ0jaWyOeelc3lvSLTmGUZI2Lyq/N7/n\nyyS9JalX3nebpDF531G57FygS36f1xdeM/8cKmmXoliukrS3pPaSzs+vO07SDxv4DA7J+8dKurae\n/Ufma4zN72XRXL5P/r2OlfRILltb0tM5znGSVs+XuS3/zirWop070XXRReY+337TtXjhtf+xTK/u\nc48ZuO16X6vBSeJ7O2zATSPHzHO9fQfMmwRX7dN77vOdtlybCW9Pae63YXXsvOtuXH/dNQBcf901\n7LLb7gDMmDGDzz//HICrrriMzbfYku7duzd4nVpTzcmuJZugVge+HxFHSroR+B5wHXAN8OOIeFjS\nWcBg4KeFkyLiOUmnA/0j4jgASYsBT0XEiXn7xYg4Kz+/FtgVuL2RWI4GZkbENyWtC8zbqyDZHLih\naHsx4MmI+JWk84Ajgd8AfwH+FBGPSeoDjAS+md/LAxHxW0kDgCOKrnV4REzLSX+UpFsi4lRJx0VE\nv3piGQbsC9wpqROptnt0vuYHEfFtSYsAj0u6JyLeKJyYa6i/BjaLiKmS6rvr/s+IuDQf/5t83b8C\npwM7RsQkST3ysT8C/hIR1+dYCrXr8cC36/tF5oR+FAAdu9Z3SJuw1JLdGPbHIwHo0L49w/41mnv/\n/RKXn30I6665AhHBW+9M48e/+eqfxRYbrMbEd6fz5qT357ne9767AXv8+OKvlR2931Zss/FafDF7\nDjM+nMmRp13Tsm+qxhx28AE8+ujDvD91Kmuu2odf/nowJ5x0CoMO3J9rr7qCFfusxNXXDwXgPy+/\nxA9/cBiS+Gbfvvztkssavc6gw45o6GWrU2XmsZK0ZLJ7IyKey8/HACtLWhzoEREP5/KrgZtKuNYc\n4Jai7W0knQwsCiwBvEDjyW4r4AKAiBgnqaF+xcsCxV+7PwfuKHoP383Ptwf6Fn3D6S6pK7AFsGd+\nnbslTS+61vGS9szPVyR9GZj3r+VX/gX8JSe0AcAjEfGppB2AdSXtnY9bPF/rjaJztwVuioipOZZp\n9Vx/nZzkegBdSQkb4HHgqvwF5Z+57AngV5JWICXJV/N150j6XFK3iPio+OIRMQQYAmnoQSPvs6ze\nnPQ+G+937jzlRzSSkB4d8yrfGfSHevf13e2MecpOOv+WeQ+0ZnPltf+ot7wwpKDYxptsynPjX56v\n69SSSq21laIlk91nRc/n0HDTYSlmRcQcAEmdgYtINb+3JZ0BdM7HzearptnO81ylaZ/WOe+L+Gog\n4hy++n21AzaJiFnFJzf0D0XS1qQEuWlEzJT0UFPxRcSsfNyOpPtiQwuXI9WMRzZ0bomuAvaIiLGS\nDgW2zq/7I0kbA7sAYyRtGBH/kPRULrtL0g8j4oF8nUWAWfNc3cwqS5UPKm/VDioR8QEwXV/1sDwY\neLiRU+pTSBJTc21q76J9bwIb5ufF5Y8ABwBIWgdoqIfAS8BqJcRwD/DjwobSfUZItaJ9c9kOQOF+\n5OLA9Jzo1gI2KbrWF5I6NvA6w4DDgC2Bu3PZSODowjmS1sjNvMUeAPaRtGQ+pr5mzG7AO/k6c++7\nSVo1Ip6KiNNJtdwVlXqnvh4RF5Duaa6bj10SmBoRXzQQv5lViLR4a2mPSlSO3piDgPNzU2I/4Kz5\nOTkiZgCXku4XjQRGFe3+PSkRPAv0Kiq/GOgq6aX8evP2LEjuJNdwmnA80D931niRdE8L4ExgB0nj\ngX2Ad4GPSImqQ379c4Eni641BBhX6KBSxz3Ad4D7cu9HgMuAF4Fn8uv8nTo19Ih4ATgHeFipU88f\n67n2acBTpARd3K5zvqTn87X/DYwlJfDxkp4D1iHddwXYhvQ7M7MqIJX2KP16aq80hOyOvL2K0rCy\nCZKG5T4ASFokb0/I+1cuusYvcvl/JO1YVD4gl02QdGqTsXi6sK/kziMPApsXmk3n8/xFgDkRMVvS\npsDFDXQ+qQqS/gmcGhGvNHZcW58uzOZVSdOFWbKw04V1XmaNWGnQX0s69pXzBpT0WpJOAPoD3SNi\n10JfgIgYKukSYGxEXCzpGGDdfBtlf2DPiNhPUl9Sp8GNgOWA+4A1CmGQ+lFMJFV6vh8RLzYUS1sd\nZ1cWEfEpqUfl8gt4iT6knpZjSR1ijmyu2Nqa/I3stqYSnZlViBJrdaXW7HKHtl1IrVEo3RDcFihM\nIXQ1sEd+PjBvk/dvl48fCAyNiM9yj/MJpMS3ETAhIl7PrV5D87ENqvjZL5rbwnT8yL0U12/GcNqs\n/A/MfejNqoRgfu7H9ZJUPKB0SO6BXezPwMmk/gEASwIzImJ23p7IVxWL5YG3AXLL2Af5+OX5+m2f\n4nPerlO+cWMBO9mZmRkwX8luamPNmJJ2BSZHxJjcG73snOzMzGxuM2Yz2RzYXdLOpB703UmTcfSQ\n1CHX7lYAJuXjJ5HGH09Umm93cdI45EJ5QfE5DZXXy/fszMwsLWjQTNOFRcQvImKFiFgZ2J80s9SB\npA6AhWFhg/hqesYReZu8/4E8xnkEsH/urbkKaQKNp0kdUlbPvTs75dcY0VhMrtmZmRnQKvNengIM\nzbM3PQtcnssvB66VNAGYRkpeRMQLuQfni6RJQ44tmmDkONLws/bAFXnIVYOc7MzMDGjWZsy5IuIh\n4KH8/HVST8q6x8wijU2u7/xzSOOG65bfBdxVahxOdmZmBlT3dGFOdmZmhjRfvTErjpOdmZkBLdOM\n2VY42ZmZGeBmTDMzqwFVnOuc7MzMjKpfz87JzszM8qDyckfRcpzszMwMqNyFWUvhZGdmZoCbMc3M\nrNo170TQbY6TnZmZzZ0Iulo52ZmZGeBkZ2ZmNcAdVMzMrLr5np2ZmVU7tc56dmXjZGdmZoBrdmZm\nVgPaVXG2c7IzMzOgRmt2kro3dmJEfNj84ZiZWTlI0L5Ge2O+AARprGFBYTuAPi0Yl5mZtbKa7KAS\nESu2ZiBmZlZeVZzraFfKQZL2l/TL/HwFSRu2bFhmZtaaRB5+UMJ/lajJZCfpQmAb4OBcNBO4pCWD\nMjOz1tdOpT0qUSm9MTeLiA0kPQsQEdMkdWrhuMzMrDXJg8q/kNSO1CkFSUsCX7ZoVGZm1qpEdffG\nLOWe3d+AW4Deks4EHgN+16JRmZlZq5NKe1SiJmt2EXGNpDHA9rlon4gY37JhmZlZa6v1ZkyA9sAX\npKbMknpwmplZ5ajkWlspSumN+SvgBmA5YAXgH5J+0dKBmZlZ62onlfSoRKXU7A4B1o+ImQCSzgGe\nBX7bkoGZmVnrqtREVopSmiTf4etJsUMuMzOzKiGab5ydpM6SnpY0VtILuXMjklaR9JSkCZKGFYax\nSVokb0/I+1cuutYvcvl/JO1YVD4gl02QdGpTMTU2EfSfSPfopgEvSBqZt3cARjX9ds3MrGI07zi7\nz4BtI+JjSR2BxyT9CzgB+FNEDJV0CXAEcHH+OT0iVpO0P6nH/36S+gL7A2uTbqXdJ2mN/Bp/A74L\nTARGSRoRES82FFBjzZiFHpcvAHcWlT85f+/ZzMwqQXPluogI4OO82TE/AtgWOCCXXw2cQUp2A/Nz\ngJuBC5Uy70BgaER8BrwhaQKwUT5uQkS8nuLW0Hzs/Ce7iLh8/t6emZlVsvmo2fWSNLpoe0hEDKlz\nrfbAGGA1Ui3sNWBGRMzOh0wEls/PlwfeBoiI2ZI+AJbM5cUVrOJz3q5TvnFjATfZQUXSqsA5QF+g\nc6E8ItZo8CQzM6sohXt2JZoaEf0bOyAi5gD9JPUAbgXWWqgAF1IpHVSuAq4k/S52Am4EhrVgTGZm\nVgYtMfQgImYADwKbAj0kFSpZKwCT8vNJwIoAef/iwPvF5XXOaai84fdWQqyLRsTIHPRrEfFrUtIz\nM7MqITVfspPUO9fokNSF1JHkJVLS2zsfNggYnp+PyNvk/Q/k+34jgP1zb81VgNWBp0mdJFfPvTs7\nkTqxjGgsplLG2X2WJ4J+TdKPSNmzWwnnmZlZBWnGYXbLAlfn+3btgBsj4g5JLwJDJf2GNF670Dfk\ncuDa3AFlGil5EREvSLqR1PFkNnBsbh5F0nHASNIMX1dExAuNBVRKsvsZsBhwPOne3eLA4aW/ZzMz\nqwTNNfQgIsYB69dT/jpf9aYsLp8F7NPAtc4h5Z665XcBd5UaUykTQT+Vn37EVwu4mplZlaniCVQa\nHVR+K3kNu/pExF4tEpGZmbU6UbnzXpaisZrdha0WhVW19b/Zh8ef8j8nszZN0K6KF29tbFD5/a0Z\niJmZlVc1r99W6np2ZmZWxYQXbzUzsxpQxa2YpSc7SYvkyTjNzKwKVXOyK2Wl8o0kPQ+8mrfXk/TX\nFo/MzMxajQTt26mkRyUq5X7kBcCupHnKiIixwDYtGZSZmbU+qbRHJSqlGbNdRLxV58blnBaKx8zM\nyiCtelChmawEpSS7tyVtBESe5+zHwCstG5aZmbW2Wh96cDSpKbMP8B5wXy4zM7MqUsUVu5LmxpxM\nnoHazMyqkxZgrbpKUspK5ZdSzxyZEXFUi0RkZmZl0b6K2zFLaca8r+h5Z2BP4O2WCcfMzMqh5juo\nRMSw4m1J1wKPtVhEZmZWFlWc6xZourBVgKWbOxAzMysjVfcMKqXcs5vOV/fs2pGWTD+1JYMyM7PW\nJ6o32zWa7JRGkq8HTMpFX0ZEgwu6mplZZRLQoYo7qDT61nJiuysi5uSHE52ZWZWSVNKjEpWSx5+T\ntH6LR2JmZmWTemOW9qhEDTZjSuoQEbOB9YFRkl4DPiH9TiIiNmilGM3MrKVV8CTPpWjsnt3TwAbA\n7q0Ui5mZlVGtjrMTQES81kqxmJlZmRSaMatVY8mut6QTGtoZEX9sgXjMzKwsRPsardm1B7pCFQ+8\nMDMzIP2hr+Jc12iyeycizmq1SMzMrHwquKdlKZq8Z2dmZrWhVjuobNdqUZiZWVnVbDNmRExrzUDM\nzKy8qrlmV8UzoZmZWakEtFdpjyavJa0o6UFJL0p6QdJPcvkSku6V9Gr+2TOXS9IFkiZIGidpg6Jr\nDcrHvyppUFH5hpKez+dcoCbmMXOyMzOzPINKs82NORs4MSL6ApsAx0rqS1ox5/6IWB24n69W0NkJ\nWD0/jgIuhpQcgcHAxsBGwOBCgszHHFl03oDGAnKyMzMzIN+3K+HRlIh4JyKeyc8/Al4ClgcGAlfn\nw64G9sjPBwLXRPIk0EPSssCOwL0RMS0ipgP3AgPyvu4R8WReoOCaomvVa0EWbzUzsyqTZlAp+Z5d\nL0mji7aHRMSQeq8rrUyaY/kpYOmIeCfvepevFgJfHni76LSJuayx8on1lDfIyc7MzID5Gm82NSL6\nN3k9qStwC/DTiPiwuAk0IkJSqy0b52ZMMzMDRLt2pT1KuprUkZToro+If+bi93ITJPnn5Fw+CVix\n6PQVcllj5SvUU94gJzszM0vNmCU+mrxWqsJdDrxUZx7lEUChR+UgYHhR+SG5V+YmwAe5uXMksIOk\nnrljyg7AyLzvQ0mb5Nc6pOha9XIzppmZATTnKuSbAwcDz0t6Lpf9EjgXuFHSEcBbwL55313AzsAE\nYCZwGKTx3pLOBkbl484qGgN+DHAV0AX4V340yMnOzMyA5psjMiIea+Ry88zOlXtUHtvAta4Arqin\nfDSwTqkxOdmZmdnccXbVysnOzMzm3rOrVk52ZmYGVPfcmE52ZmYG1OiqB2ZmVjtSM2b1ZjsnOzMz\nA1yzMzOzqifkmp2ZmVU71+zMzKyqSdC+irOdk52ZmQHVXbOr5jGEZs3mhz84nD7LLcWG/b6aneiW\nm29ig/XWZtFO7RgzenQjZ1s5XPDnP7HBemuzYb91OOSg7zNr1iy223pLNt6wHxtv2I9V+izHPt9r\ndL3PmqMS/6tETnZmJTh40KEMv+Pur5WtvfY6DL3xn2yx5VZlisoaMmnSJC762wU8/uRoxjw3njlz\n5nDTsKHc/9CjPDXmOZ4a8xwbb7Ipe+yxV7lDbTPS4q2lPSqRmzHNSrDFllvx1ptvfq1srW9+szzB\nWElmz57Np59+SseOHfl05kyWXW65ufs+/PBDHn7wAYZcdmUZI2x7KrXWVgrX7Mys6iy//PL89Gcn\nscY3+rDKisvSvfvibP/dHebuv334bWy97XZ07969jFG2Pe2kkh6VqGKSnaTjJb0k6foWuPbWku5o\nYN/6ki5v5te7S1KP/DimqHw5STc352vVed2rJO3dTNfqLenupo80a33Tp0/njtuH89Krb/D6f//H\nJzM/4Ybrr5u7/8ZhN7Dvft8vY4RtT7U3Y1ZMsiMt1PfdiDiwuFBSSzfF/hK4oDkvGBE7R8QMoAfp\nfRXK/xcRzZKMWlpETAHekbR5uWMxq+uB++9j5ZVXoXfv3nTs2JE99tiLJ5/4NwBTp05l9Kin2Wnn\nXcocZVtTaveUysx2FZHsJF0CfAP4l6SfSTpD0rWSHgeulbSypEclPZMfm+XzvlZjk3ShpEPz8wGS\nXpb0DFDvXWpJ3YB1I2Js3i687hOSXpV0ZC6XpPMljZf0vKT9cvmykh6R9Fzet2Uuf1NSL9Kqvavm\n/efn9zE+H/OkpLWLYnlIUn9Ji0m6QtLTkp6VNLCB2E/JsYyVdG49+0+XNCrHNSQvbV+oQb8oaZyk\nobnsOznG5/JrdsuXuQ04sO61zcptxRX78PTTTzJz5kwiggcfuJ8110r3WG+95WZ22nlXOnfuXOYo\n2xiloQelPCpRRXRQiYgfSRoAbBMRUyWdAfQFtoiITyUtSqr1zZK0OnAD0L+h60nqDFwKbEtaBn5Y\nA4f2B8bXKVsX2ARYDHhW0p3cY9vdAAAe10lEQVTApkA/YD2gFzBK0iPAAcDIiDhHUntg0TrXOhVY\nJyL65bhWLto3jLRk/WBJywLLRsRoSf8HPBARh0vqATwt6b6I+KTo/e0EDAQ2joiZkpao571dGBFn\n5eOvBXYFbs8xrRIRn+XrA5wEHBsRj0vqCszK5aOB3zTwu6sqhxz0fR59+CGmTp3KqiuvwGmnn0nP\nJZbghJ/+mKlTprDXwF1Yd71+3H7XyHKHasBGG2/MnnvtzaYbbUCHDh1Yb731OeLIowC46cahnHTy\nqWWOsG2q0DxWkopIdg0YERGf5ucdgQsl9QPmAGs0ce5awBsR8SqApOuAo+o5bllgSp2y4fl1P5X0\nILARsAVwQ0TMAd6T9DDwbWAUcIWkjsBtEfHcfLy/G4F7gMGkpFe4l7cDsLukk/J2Z6AP8FLRudsD\nV0bETICImFbP9beRdDIpAS8BvEBKduOA6yXdRqq5ATwO/DHfL/1nREzM5ZOB5aiHpKPIv9MV+/SZ\nj7fdNl1z3Q31lg/cY89WjsRKddrgMzlt8JnzlN9z/0OtH0wFSPfsqjfdVUQzZgM+KXr+M+A9Us2q\nP9Apl8/m6+9xftstPq3nnGhi+6sdEY8AWwGTgKskHVLqC0fEJOB9SesC+/FV7VPA9yKiX370iYiX\nGrxQPXLN9iJg74j4FqmWW3ifuwB/AzYg1VA7RMS5wA+ALsDjktbKx3Ym/Y7qi39IRPSPiP69e/We\nn/DMrEyquRmzkpNdscWBdyLiS+BgoH0ufwvoK2mR3CS3XS5/GVhZ0qp5u6FuWS8Bq9UpGyips6Ql\nga1JtbdHgf0ktZfUm5Tgnpa0EvBeRFwKXEZKIMU+ArrRsGHAycDiETEul40Eflx0j239es67Fzgs\nN+9STzNmIbFNzc2Se+fj2gErRsSDwCmk32tXSatGxPMR8bv8fgvJbg3mbeY1swrlDipt30XAIElj\nSX+IPwGIiLdJzYHj889nc/ksUhPbnbmDyuT6LhoRLwOLF3XIgNTM9yDwJHB2RPwPuDWXjwUeAE6O\niHdJyXCspGdJtbO/1Ln++6Sa0nhJ59cTws3A/jn2grNJzbbjJL2Qt+vGfTcwAhgt6TnSPbfi/TNI\ntbnxpOQ5Ku9qD1wn6fn8u7ogH/vTHOM44AvgX/n4bYA764nbzCpQNdfsFNFgK5wBkn4GfBQRl+WO\nMR9HxO/LHFabkDvhDIyI6Y0dt+GG/ePxpzx3pFlL6tJRYyKiwY55Tfnmt9aPa4Y/VNKxG63aY6Fe\nqxyqpWbXki4GPit3EG1Nbq79Y1OJzswqiEp8VKBK7o3ZKnKT57X5+RnljabtyIPKb2vyQDOrCCmP\nVWgmK4GTnZmZQQVPBVYKJzszM0uc7MzMrLpV7rCCUjjZmZkZULnDCkrhZGdmZpXc0bIkHnpgZmYA\nSCrpUcJ1rpA0ubCKSy5bQtK9SivG3CupZy6XpAskTcirrWxQdM6gfPyrkgYVlW+YV3WZkM9tMign\nOzMzA5p1BpWrgAF1yk4F7o+I1YH78zbATsDq+XEUaWxzYZrDwcDGpAn3BxcSZD7myKLz6r7WPJzs\nzMwMaL4x5XkS/LqrrQwErs7Prwb2KCq/JpIngR55WbMdgXsjYlqevOJeYEDe1z0inow0Bdg1Rddq\nkO/ZmZnZ/N606yWpeA7AIRExpIlzlo6Id/Lzd4Gl8/PlgbeLjpuYyxorn1hPeaOc7MzMDJivGVSm\nLszcmBERklp1YmY3Y5qZWarYteyqB+/lJkjyz8JqM5OAFYuOWyGXNVa+Qj3ljXKyMzMzoMWT3Qig\n0KNyEDC8qPyQ3CtzE+CD3Nw5EthBUs/cMWUHYGTe96GkTXIvzEOKrtUgN2OamRnQfBNBS7qBtJ5n\nL0kTSb0qzwVulHQEaWHtffPhdwE7AxOAmcBhABExTdLZfLXe5lkRUej0cgypx2cX0vqahTU2G+Rk\nZ2ZmQPPNoBIR329g13b1HBvAsQ1c5wrginrKRwPrzE9MTnZmZgZU9wwqTnZmZpZUcbZzsjMzMy/e\namZmNcCLt5qZWU1wsjMzs+rmxVvNzKwGePFWMzOratW+eKuTnZmZAZS0MGulcrIzMzPAzZhmZlYD\nqjjXOdmZmRmwcCsatHlOdmZmllVvtnOyMzOzuYu3VisnOzMzAzxdmJmZ1QDPoGJmZtWvenOdk52Z\nmSVVnOuc7MzMLHVOcQcVMzOrep4uzMzMql71pjonOzMzy6q4YudkZ2Zm4MVbzcys6lX7DCrtyh2A\nmZlZS3PNzszMAGhXxVU7JzszM/MSP2ZmVv2Ehx6YmVktqOJs52RnZmZAda964N6YZmYGfDU/ZlOP\n0q6lAZL+I2mCpFNbNvKmOdmZmRnQfMlOUnvgb8BOQF/g+5L6tmz0jXOyMzMzoDCHStP/lWAjYEJE\nvB4RnwNDgYEtGnwTfM/OWtwzz4yZ2qWj3ip3HC2gFzC13EHYfKnmz2ylhTn52WfGjFy0k3qVeHhn\nSaOLtodExJCi7eWBt4u2JwIbL0x8C8vJzlpcRPQudwwtQdLoiOhf7jisdP7MGhYRA8odQ0tyM6aZ\nmTW3ScCKRdsr5LKycbIzM7PmNgpYXdIqkjoB+wMjyhmQmzHNFtyQpg+xNsafWSuIiNmSjgNGAu2B\nKyLihXLGpIgo5+ubmZm1ODdjmplZ1XOyMzOzqudkZ1blpGpeuKXy+PMoDyc7syojaUVJh0taWlL7\niIg8fZOVkaTlJa0d7ihRFu6NaVZ95gBbAasAy0g6NiI+lyT/oS2rDYGfShoBvBcRN5Q7oFri3phm\nVURSu4j4Mo9tWhIYDGwCfDciphT2lzfK2iVpTeCbwFHAaxHx4zKHVDOc7MyqRKHmJqkP8HahFifp\nL8B3gE0j4lPX8FpX0efSOSJm5bIlgfuAJyPi6PJGWBt8z86sSuQ/qLuSZqr4hqQOufwnwFPAdZI6\nONG1nqJEtxtwvaSu+T7q+8DWwBqSjipvlLXByc6sSkjaHPgdcHhEvAZ0ldQz7z4OmACsVq74alFO\ndAOAs4ELI+JjSGvkRMQHwP8By0lq716aLcvJzqx6dAOuA3pLOpbUTHampH55fwfS/TtrXd8GzgTe\nkLQPMELSfpI6A6+TPhP30mxhTnZmFapQE5D0HUmrAZOBb5A6pXwM/JL0/3iPiPgC+C0wrdC8aS2j\n6HPpm59/CBxK+iKyEvAsaSHTRSPiDeB80jp71oLcQcWsguV7Qb8DfhIR9+Zmyy8j4gNJa5FWiD46\nIp7Ix7tzSiuQtAtwOnBYRLwoaRNgckS8LmlV4AbgoIh4RVJHgPyFxFqIk51ZhZK0HHA7cGBEvJy7\ntfcAnga2JyXBMyNieBnDrDmSvgXcDHwvIsZLWhz4NI913BM4C/i1P5fW5eYMswpTVDvrTGq63FDS\n8aTFMvsDRwAvAUdExLOuzbWOojGMfYDRwOeSfgUMAJbKtbtOwPER8aA/l9blmp1ZhSjqxr5CREzM\nZT8G+gHDI2KEpB8Cq0XEz8sabA0p+lwWjYiZuez+vPsfpFren4AREXFbueKsda7ZmVWAoplRdgF+\nJelRYApwUdEf2C2A44GflDHUmlKU6AYAgySNBu6OiO2KPrNvARsDF5c32trm3phmbVjunk7+o7kF\nqUflwUBX4PvAeXnC5xWA84BTIuK+sgVcIwo9LnOi2x74PXAhsBtwjqT9iz6z4cCpETGqfBGbk51Z\nG5WnlPq5pG1z0ZKkRLcaqaYwGFie1OtvOrBXRNxRjlhriaRlgL0k9c7DOLYB9gO6kDoIjcn79wCe\nJH0ut3vQeHm5GdOs7eoOLA5sL+nDiBiel+o5ntSl/XlJ3wN6AitFxIvlDLaGbA4cROpscgtpdpSu\npPty20bENEljgV2Bf0fEc5BqgWWK13CyM2uT8r2gNyRdAPwA2EfSIhHxeG6y3C+Pz+pL6nXpRNdK\nIuKWXEvbizT117+AIPXCXCaPdZwC/CkiJpcvUivm3phmbUxRp4fNganATOBIYBHgCuBT4BrgC2BI\nRNxUtmBrSNHnsirwBinZ7Q6MjIjrJR0HHAN8CZwWEbeWMVyrw8nOrA3KM6OcAZwcEffn6cAOISW8\nqyLiJUndIuIjj9dqPXlViRNJM9aMk3QAsDNwJ3Aj6R5qhzxTij+XNsTJzqyNkdSLdC/opIgYVVSj\nWI1Uw+sInBURM8oaaI3J068NA35Q3LNS0v7AvsCtwHVOcG2T79mZtT0dSLOjvJO3OwKfA/8ldXHv\n6URXFksBbxYSnaROEfF5RAyVNCvvc6Jrozz0wKyNiYh3gUdJnVJ65TkVv0OaNX9WRLxS3ghr1n+B\nWZI2ywPGP5e0taSfRcRthV6X1jY52Zm1TQ8Cy5JWFz8GuAy4MiI+Km9YNe090gK4u5DGP34X+Dsw\nrqxRWUl8z86sTIruxXWsb3kXSd8AtiUNVh4fEQ+2epAGfO2zWhLYEdiK1Lx8qwfyVwYnO7MykrQz\nKaF9DFwPvJZnzrcyKEpqWwErAB8Wklnd3pWSOkfELPe6rAxuxjQrk3wf7nfAVcCBpJlR2pczplqX\nE90OwCWkTkEjJB1WnNCK5sWcVTinbAFbydwb06x8tgBOIU0LNg04LyK+yDOlfFbe0GpPTmKLAoeR\n5rrsBowlrWIwN6E5uVUmJzuzVpZn4PiI1NnhaGBpYJ+IeFvSQaSOKeeXMcRa1TPPa/kMcCiwGbB3\nRLwj6Qjg5Yh4vKwR2gJzM6ZZK1HSDTiLtGrBU6RJnC8G3pfUDzgZGF++KGtP/lxWAq6XtARperb9\ngCMj4jVJ6wInkDqkWIVyBxWzFiapQ0TMLto+gDTl1LbA9sAAYEXSQPI/5dUN3OmhhdXT4eRq4P2I\nOEHSVaTPYw6wFnBmRIwoT6TWHJzszFqIpBUiYmJ+viawGGkIweeSfgc8GRG35vXRZgOdI2KiE13L\nktQ9Ij7Mz5cGZkTEZ5L6AL8Efp7nHN2KNKnzRxEx1p9LZXMzplnL+T9J6+WOD0cAxwI35LkvJ5Nm\nzSci3o2IqYXE6D+oLUfSosAQSctJWhwYAQyW9KOI+C+wBPAjgIh4JCIei4ixedufSwVzzc6sBUla\nhTRp88F5/bk/kVazfol07+7wiLi6nDHWmjwwfAlgFWAUsCFphYlHgPeBo4CBEfFyuWK05ueanVkz\nktRVUvf8/JsR8QbQX9KwiPgiIo4jTf31H9ICn6+VMdyaIalL0WYHYDlgCLB9RNwHbE0a2L8MsHqr\nB2gtzjU7s2YkaVPgt8CVpBrC7hHxvqSngHciYo+iY7tGxMe+F9SycjPy7sDawOOkcXTHA5sC55E6\nBV1VdPzqEfFqGUK1FuRkZ9bMJF0LfB/YPyJuLip/gjT91I55u52nBmtZknpHxBRJ7YDRwKqk2lxh\nmZ4dSF9OLo6Iy3JZYcowfwmpIm7GNGt+d5Fqdn+UtHKhMCI2BRaVtGHedqJrQZLaAxdIWp00Ddsz\nwPOkFd8BiIh7SD0wj5e0fHGCc6KrLq7ZmS2koprA2qSxWc9FxBxJZwI/AFbLj/4RcWU5Y601kjoB\nK5NmqDlHUgfgNmByRByeV3/vRZodxQviVjHX7MwWUk50uwI3AccAj0paPyIGkyZ5Hgf8A5havihr\nS2GyZuAL4APgcEm/zYP7jwD6SLofuBno4ERX/VyzM1tIktYn9ewbCPQnJbhHgbMjYrSkDYBPI+Il\n3wdqeUU17Z2A1SPigjxw/25gZESckps4jwWejYhHyxqwtQonO7MFUGfJl8WANUjj584DdiIt3bMl\nMCginihboDVK0gDgD8AxEfFwLusNDAeeyUNArIa4GdNsAeSaw5aSDoyITyLiWVKt7t6ImAr8k9Rs\n+UFZA60xeVLnRUjNyadHxMOSdpJ0Gum+6a7AdyStVdZArdW5Zmc2H4qayDYFfkH64/mjiBgi6bvA\nr4F/A98FfuYmsvKQdCJpHF13vhq4/3lE/MTrBdYmr2dnNh9yotuKNAvKINLciudI+gK4htQbczfg\nNCe61lH0BaQ/0Ad4E7iHtFTSu3kS522AsyT1xLXtmuRkZzb/+pBWr34CeELSf4D7SDWH6yXd4UHJ\nrSf/rncE/k7qEXsWqcPQ8Ih4Kye6C4GTI2J6GUO1MvI9O7MmFHVjL3gHWFZSZ0ntcw3uGuAPknb3\noOTWk+/RLUYaKH50RPycNLTgW8A2kjoDGwEnRcSd9XyWViN8z86sEXW6sa9LaqY8B7ge+BC4CFiK\n9Md2PPCNiDiyXPHWKkl/JX0J+UNem25n4DRgK6B9RMwqa4BWdq7ZmTUiJ7odSAnuXtJaZ6cDB5IG\nLB9NmlvxD8ArQKc8D6O1kELtTNJqkr6dt58AugCb58PeJK0q0cmJzsD37MxKsROpaWxp4A3gqjwT\nxzGQVr4m/ZE9HTjIc162nKKa9s7AX4D3gEnAjUBH4BBJPwFWIg3q/6R80Vpb4mRn1rQPSLNtrAIc\nGhGvSzoIWCwi/g7MIY3hOjgini9jnFVLUqeI+DwnujVJXzT2jIjxkk4hjXG8htS0/E3gvYgY505C\nVuDmFrMiRU1k60jqkxf9vIe0ZM9fI+IVSd8mzZT/CkCuPVwYEePKFXc1k9SVtILEUpJ6kNYJXJO0\nACsR8TtSTe7oiJgYEfcWPgsnOitwzc4sK2oi2xa4DngMeB/4DSnZnSNpL9If2lMi4sHCOf6j2qLm\nAJ+Q5hf9SNINpL9dW0l6PyLGkD6vnQs1wHIGa22Te2Nazaszz+UWpNlPhpOaL/ck9cL8BTAL6Aos\nkmt4biJrJZL+DPSJiL3y9ibAXsCGpC8lewK/iojbyxeltWVOdlbTJC0N7BoRl+ftR4G1gJUiYqak\nNUgzomwC/D4inipftLWnzheRq4CXcrMleTWJHwCLkgb5Dy1boNbm+Z6dGRwgaWuAiNiS1G392rz9\nCnAn8DSpZmctLA8EB+YO/SgMBL8eWFnSSXnfM8DVpPF1/ST1bfVgrWI42VnNktQuIt4DLgf65o4Q\nRMS3gZUkDcvbLwN/i4ix5Yu2NkhaHLgi3zet63HSXKR9JV0rqUOuaf8D+ByY3IqhWoVxM6bVPEnf\nAU4EfhMRTxeVvwy8HBF7lC24GiNpCeAAYHvg3Ih4Mpe3i4gv84D9xYA/ArOBl0hfVr5wxxRrjJOd\n1ZQ8l+Wc/Lz4ftDxwOGkwePjC0vASNoqIh4pW8A1KK8qMYg0tGBw4QtI3Q5BklYBBEyJiI/KEqxV\nDCc7qxmSliNN73VxRPwvl7UrzHgi6Wek2VKuA14p1Cqs9UjaBfg96V5cP1Lv13Mj4rGiY+Z+Zmal\ncrKzmpEHJN8APANcFBGTcnlxbW8nYA1SV/a/A/+KiBllCrnmSPoj8HBEDJe0PLAHMAA4x18+bGE4\n2VlNyJ0ZZkvaDjgfeIG0kvjUvP9rtQVJywIrAs/5XlDLk9QjImZIuhjoEhGH5vJvA38DpgP7AR94\nbKMtCPfGtJqQE93OwBnAFcDGwGBJy+T9XxZNFdYuIt6JiKed6FpeHst4mqQNSQuvdpF0Wt79BfAy\n8NOImOFEZwvKNTurepI6kL7YXQU8GBGX5i7u15FmSTkhItxtvUzyF45fAJ8C/wI+Bv4EfAT0JSW6\n4eWL0KqBk51VPUmdI2JWri18Rrpf97GklUhd1/8MnOV1z1qXpH6kIQMv5IT3UyBIX0JeAlYFiIhX\nPTWbLSw3Y1pVk7QaMCb3xHyYNO3XunmWjkVIKxrc7kTXuiR1AnYhrWbQNyLeBS4g9cD8P2DTiHg1\nIl4Fr15gC8/JzqpaREwgNY3dCowFbgGOB4aSZuO4JCKeKF+EtaNo2i/yvdALgEeAsyWtk4eDDAXa\nkxZlNWs2bsa0qiTpG8CMiJiWt88BdgS2Iy0ZsyYw21OAtY6i5ZN2Ja3q3hE4jfRZ/Bz4HnAlcBjw\nk4h4tGzBWlVysrOqIqk9qXlyOPAk8KeihDeM1OFhp4iYWL4oa5Ok3Ui9YY8iDScA2C4iPpF0JKkJ\n8/aIuLtMIVoVczOmVYWiJrKIiJmkmVL6AcdIWjLvuxN4F1i59SOsbXnF962AQ0njFz8BJgHPSuoa\nEZeSxj3eXdzcadZcXLOzilfURLYdsCvwBHAvaZ2zS0gDyCcD+wOHRsSLZQu2hhRmppE0AFgPGAL0\nJN2X2y0i3pM0mXR/bj1I4x3LFrBVNdfsrOLlRLcj8FfgKeBnwLmkeRV/AEwhLch6jhNdy5O0kqSV\ncqJbhzSG7q6ImA7MBMYDi0naDLgMODoivnSis5bUodwBmC2oohrdssDuwECgD9CdNDD5ROAvEfGH\nuueUJeAaIGll4G7gcElB+rLRnpTgIH3Bnk1KgAOBgyPiMX8u1tLcjGkVTdL2QGfgedJyL9eTmjJ7\nk4Yc3AmcGRHvly3IGpHvtX2PNJbxAtIkzp8CuwG3A//InVFWALoAi0XEc+WK12qLmzGtokhaOs+G\nX5iB4whgckS8BSwOdMrNZSLNwnGRE13ryDWz+4ADSWMa784dT24D1gH2yZ1RJuYB40501mqc7Kxi\nSFodeIA0UXA3UkeUThHxdO4MMRaYJOlpUk1iSES8XMaQa9GHwLOkeS03z2VX57LNgP3yauNmrcr3\n7KySbEqaAX8ZoBdpyZebJO0SEXcCRMQekrYhrV493veCWlfuZLKzpD7AfZJ6RsQfJV1D+nvzhDui\nWDn4np1VDEm9SCsXbAocnhf43Ie0ZM/+hYRnbUNuZr4euDYizi13PFbb3JxglWZR4CFgdUlLRMRN\nwCHA7XmGDmsj8j25g4CjJK3sweJWTq7ZWcXIfyx7At8g/RH9EPh9RHyYa3gfRsTIcsZo85LULSI+\nKnccVtuc7KxNK77nVjSurh1pQucdSRMJ/19EfFD3eGsb/JlYW+BmTGuTipq85v4bzYmuQ+7gcF9+\ndAaWKj6mVQO1JvkzsbbANTtrc4pqcNsDBwOvA69FxHV5f2HOxXZAT4+jM7OmuGZnbU5OdN8hzXX5\nEGl2/GMlnZj3z8kJ8UsnOjMrhcfZWVu1AnBpRFwJIOkp4HxJd0fEC24aM7P54ZqdtQn1dEvvQupx\nWfACaSkYJzkzm29OdtYmFJouJR0jqW9EXAY8Jel+SUsA/YF1gY7ljdTMKpE7qFhZFXVG2Zg0E8qL\npDXPHiPNvnEeaWXxJYHfRsSIcsVqZpXLyc7KTtJGwFnAyRExTtL3ScvEjIuIy3Ovyx4RMc1jtsxs\nQbgZ09qCHsD2wHfz9k3A48Amkn5CWq5nOnjMlpktGPfGtLKLiHsk7QX8VtL/IuIGSTeTVrgeGxFz\nyhyimVU4JztrEyJihKTZwNmSOkXE1cAN5Y7LzKqD79lZmyJpd+BcUrPmu177zMyag5OdtTmSekfE\nlHLHYWbVw8nOzMyqnntjmplZ1XOyMzOzqudkZ2ZmVc/JzszMqp6TnVmZSZoj6TlJ4yXdJGnRhbjW\n1pLuyM93l3RqI8f2kHTMArzGGZJOKrW8zjFXSdp7Pl5rZUnj5zdGs7qc7MzK79OI6BcR6wCfAz8q\n3qlkvv9fjYgREXFuI4f0AOY72ZlVIic7s7blUWC1XKP5j6RrgPHAipJ2kPSEpGdyDbArgKQBkl6W\n9AywV+FCkg6VdGF+vrSkWyWNzY/NSIP3V821yvPzcT/X/7d3N6FVHWEYx/+PUWvQKN2Uopv4bW1R\nMUQKhVKKBPyCbASDImIwNgtRhEKh6ULorjsXYq1CFwVRaF0VCdKFX0RNUKMLNWJFNwWzCtbYjbwu\nziscbw056EI9PD+43HvnzJn3zN28zMxlRhqUdEPSgVJb30sakXQBWDpZJyTtynaGJf3eMFpdK2ko\n29uY9Zsk/VSKvftNf0izMic7s3eEpKnAOuBmFi0GDkXEp8AToA9YGxGrgSFgv6QZwC/AJqAN+HiC\n5g8CZyNiJbCa4jDc74B7Oar8VlJHxlwDrALaJH0pqQ3YkmXrgfYK3fkjItoz3i2gu3StNWNsAA5n\nH7qBsYhoz/Z3SZpfIY5ZJd4b0+zta5Z0PT+fB44Bc4EHEXEpyz8HlgMX81D36cAAsAy4HxF3AST9\nBvS8IsbXwHaA3Fh7TNKHDXU68nUtv8+iSH4twKmIGM8YVc4U/EzSjxRTpbOA/tK1k7kN3F1Jf2cf\nOoAVpfW8ORl7pEIss0k52Zm9fU8jYlW5IBPak3IRcCYiuhrqvXTfGxLFAbk/N8TY9xpt/Qp0RsSw\npB3AV6Vrjds2RcbeExHlpIik1teIbfY/nsY0ez9cAr6QtAhA0kxJS4DbQKukhVmva4L7/wJ6894m\nSXOAxxSjthf6gZ2ltcB5kj4CzgGdkpoltVBMmU6mBfhH0jRga8O1zZKm5DMvAO5k7N6sj6QlkmZW\niGNWiUd2Zu+BiBjNEdJxSR9kcV9EjEjqAf6UNE4xDdryiib2AkckdQPPgN6IGJB0Mf/afzrX7T4B\nBnJk+S+wLSKuSjoBDAOPgMEKj/wDcBkYzffyMz0ErgCzgW8i4j9JRynW8q6qCD4KdFb7dcwm542g\nzcys9jyNaWZmtedkZ2ZmtedkZ2ZmtedkZ2ZmtedkZ2ZmtedkZ2ZmtedkZ2ZmtedkZ2Zmtfcc+3u2\nHm4a1TgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a295a5320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(json.dumps(metrics, indent=4, sort_keys=True))\n",
    "plot_confusion_matrix(cm, ['no fraud (negative class)', 'fraud (positive class)'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In business terms, if the system classifies a fair transaction as fraud (false positive), the bank will investigate the issue probably using human intervention. According to a [2015 report from Javelin Strategy](https://www.javelinstrategy.com/press-release/false-positive-card-declines-push-consumers-abandon-issuers-and-merchants#), 15% of all cardholders have had at least one transaction incorrectly declined in the previous year, representing an annual decline amount of almost $118 billion. Nearly 4 in 10 declined cardholders report that they abandoned their card after being falsely declined.\n",
    "\n",
    "However, if a fraudulent transaction is not detected, effectively meaning that the classifier predicts that a transaction is fair when it is really fraudulent (false negative), then the bank is losing money and the bad guy is getting away with it.  \n",
    "\n",
    "A common way to use business rules in these predictions is to control the threshold or operation point of the prediction. This can be controlled changing the threshold value in `binarize_prediction(y_prob, threshold=0.5)`. It is common to do a loop from 0.1 to 0.9 and evaluate the different business outcomes.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.save_model(BASELINE_MODEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## O16N with Flask and Websockets\n",
    "\n",
    "The next step is to operationalize (o16n) the machine learning model. For it, we are going to use [Flask](http://flask.pocoo.org/) to create a RESTfull API. The input of the API is going to be a transaction (defined by its features), and the output, the model prediction.\n",
    "\n",
    "Aditionally, we designed a [websocket service](https://miguelgfierro.com/blog/2018/demystifying-websockets-for-real-time-web-communication/) to visualize fraudulent transactions on a map. The system works in real time using the library [flask-socketio](https://github.com/miguelgrinberg/Flask-SocketIO). \n",
    "\n",
    "When a new transaction is sent to the API, the LightGBM model predicts whether the transaction is fair or fraudulent. If the transaction is fraudulent, the server sends a signal to the a web client, that renders a world map showing the location of the fraudulent transaction. The map is made with javascript using [amCharts](http://amcharts.com/) and the map locations are taken from the previously created SQLite database.\n",
    "\n",
    "To start the api execute `(fraud)$ python api.py` inside the conda environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also run the api from inside the notebook (even though I find it more difficult for debugging).\n",
    "# To do it, just uncomment the next two lines:\n",
    "#%%bash --bg --proc bg_proc\n",
    "#python api.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we make sure that the API is on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html> <html lang=\"en\">\n",
       "<html>\n",
       "<head>\n",
       "    <link rel=\"stylesheet\" type=\"text/css\" href=\"http://fonts.googleapis.com/css?family=Fjalla One\">\n",
       "    <title>Fraud detection API</title>\n",
       "    <style>\n",
       "        .main{\n",
       "            font-family: 'Fjalla One', serif; \n",
       "            text-align: center;\n",
       "        }\n",
       "    </style>\n",
       "</head>\n",
       "<body>\n",
       "    <div class=\"main\">\n",
       "    <h1>The fraud police is watching you</h1>\n",
       "    <p align=\"center\">\n",
       "        <img src=\"https://i.giphy.com/media/81xwEHX23zhvy/giphy.gif\">\n",
       "        <!-- <img src=\"police.gif\"> -->\n",
       "    </p>\n",
       "    </div>\n",
       "</body>\n",
       "</html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#server_name = 'http://the-name-of-your-server'\n",
    "server_name = 'http://localhost'\n",
    "root_url = '{}:{}'.format(server_name, PORT)\n",
    "res = requests.get(root_url)\n",
    "display(HTML(res.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to select one value and predict the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Time': 57007.0, 'V1': -1.2712441917143702, 'V2': 2.46267526851135, 'V3': -2.85139500331783, 'V4': 2.3244800653477995, 'V5': -1.37224488981369, 'V6': -0.948195686538643, 'V7': -3.06523436172054, 'V8': 1.1669269478721105, 'V9': -2.2687705884481297, 'V10': -4.88114292689057, 'V11': 2.2551474887046297, 'V12': -4.68638689759229, 'V13': 0.652374668512965, 'V14': -6.17428834800643, 'V15': 0.594379608016446, 'V16': -4.8496923870965185, 'V17': -6.53652073527011, 'V18': -3.11909388163881, 'V19': 1.71549441975915, 'V20': 0.560478075726644, 'V21': 0.652941051330455, 'V22': 0.0819309763507574, 'V23': -0.22134783119833895, 'V24': -0.5235821592333061, 'V25': 0.224228161862968, 'V26': 0.756334522703558, 'V27': 0.632800477330469, 'V28': 0.25018709275719697, 'Amount': 0.01}\n"
     ]
    }
   ],
   "source": [
    "vals = y_test[y_test == 1].index.values\n",
    "X_target = X_test.loc[vals[0]]\n",
    "dict_query = X_target.to_dict()\n",
    "print(dict_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "{\n",
      "  \"fraud\": 1.0\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "headers = {'Content-type':'application/json'}\n",
    "end_point = root_url + '/predict'\n",
    "res = requests.post(end_point, data=json.dumps(dict_query), headers=headers)\n",
    "print(res.ok)\n",
    "print(json.dumps(res.json(), indent=2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fraudulent transaction visualization\n",
    "\n",
    "Now that we know that the main end point of the API works, we will try the `/predict_map` end point. It creates a real time visualization system for fraudulent transactions using websockets. \n",
    "\n",
    "A websocket is a protocol intended for real-time communications developed for the HTML5 specification. It creates a persistent, low latency connection that can support transactions initiated by either the client or server. [In this post](https://miguelgfierro.com/blog/2018/demystifying-websockets-for-real-time-web-communication/) you can find a detailed explanation of websockets and other related technologies. \n",
    "\n",
    "<img src=\"https://miguelgfierro.com/img/upload/2018/07/12/websocket_architecture2.svg?sanitize=true\">\n",
    "\n",
    "For our case, whenever a user makes a request to the end point `/predict_map`, the machine learning model evaluates the transaction details and makes a prediction. If the prediction is classified as fraudulent, the server sends a signal using `socketio.emit('map_update', location)`. This signal just contains a dictionary, called `location`, with a simulated name and location of where the fraudulent transaction occurred. The signal is shown in `index.html`, which just renders some javascript code that is referenced via an `id=\"chartdiv\"`. \n",
    "\n",
    "The javascript code is defined in the file `frauddetection.js`. The websocket part is the following:\n",
    "```js\n",
    "var mapLocations = [];\n",
    "// Location updated emitted by the server via websockets\n",
    "socket.on(\"map_update\", function (msg) {\n",
    "    var message = \"New event in \" + msg.title + \" (\" + msg.latitude\n",
    "        + \",\" + msg.longitude + \")\";\n",
    "    console.log(message);\n",
    "    var newLocation = new Location(msg.title, msg.latitude, msg.longitude);\n",
    "    mapLocations.push(newLocation);\n",
    "\n",
    "    //clear the markers before redrawing\n",
    "    mapLocations.forEach(function(location) {\n",
    "      if (location.externalElement) {\n",
    "        location.externalElement = undefined;\n",
    "      }\n",
    "    });\n",
    "\n",
    "    map.dataProvider.images = mapLocations;\n",
    "    map.validateData(); //call to redraw the map with new data\n",
    "});\n",
    "```\n",
    "When a new signal is emited from the server in python, the javascript code receives it and processes it. It creates a new variable called `newLocation` containing the location information, that is going to be saved in a global array called `mapLocations`. This variable contains all the fradulent locations that appeared since the session started. Then there is a clearing process for amCharts to be able to draw the new information in the map and finally the array is stored in `map.dataProvider.images`, which actually refresh the map with the new point. The variable `map` is set earlier in the code and it is the amCharts object responsible for defining the map.\n",
    "\n",
    "To make a query to the visualization end point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"fraud\": 1.0\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "headers = {'Content-type':'application/json'}\n",
    "end_point_map = root_url + '/predict_map'\n",
    "res = requests.post(end_point_map, data=json.dumps(dict_query), headers=headers)\n",
    "print(res.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can go the map url (in local it would be http://localhost:5000/map) and see how the map is reshesed with a new fraudulent location every time you execute the previous cell. \n",
    "You should see a map like the following one:\n",
    "[![map](./static/img/map.png)](https://youtu.be/KiCeeJAlgJU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load test\n",
    "\n",
    "Once we have the API, we can test its scalability and response time. \n",
    "\n",
    "Here you can find a simple load test to evaluate the performance of your API. Please bear in mind that, in this case, there is no request overhead due to the different locations of client and server, since the client and server are the same computer. \n",
    "\n",
    "The response time of 10 requests is around 300ms, so one request would be 30ms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "num = 10\n",
    "concurrent = 2\n",
    "verbose = True\n",
    "payload_list = [dict_query]*num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR:asyncio:Creating a client session outside of coroutine\n",
      "client_session: <aiohttp.client.ClientSession object at 0x7f16847333c8>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "Response status: 200\n",
      "{'fraud': 7.284115783035928e-06}\n",
      "CPU times: user 14.8 ms, sys: 15.8 ms, total: 30.6 ms\n",
      "Wall time: 296 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with aiohttp.ClientSession() as session:  # We create a persistent connection\n",
    "    loop = asyncio.get_event_loop()\n",
    "    calc_routes = loop.run_until_complete(run_load_test(end_point, payload_list, session, concurrent, verbose))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you run the API from the notebook, you can uncomment the following two lines to kill the process\n",
    "#%%bash\n",
    "#ps aux | grep 'api.py' | grep -v 'grep' | awk '{print $2}' | xargs kill"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enterprise grade reference architecture for fraud detection\n",
    "\n",
    "In this tutorial we have seen how to create a baseline fraud detection model. However, for a big company this is not enough. \n",
    "\n",
    "In the next figure we can see a reference architecture for fraud detection, that should be adapted to the customer specifics. All services are based on Azure.\n",
    "\n",
    "1) Two general data sources for the customer: real time data and static information.\n",
    "\n",
    "2) A general database piece to store the data. Since it is a reference architecture, and without more data, I put several options together ([SQL Database](https://azure.microsoft.com/en-gb/services/sql-database/), [CosmosDB](https://azure.microsoft.com/en-gb/services/cosmos-db/), [SQL Data Warehouse](https://azure.microsoft.com/en-gb/services/sql-data-warehouse/), etc) on cloud or on premise.\n",
    "\n",
    "3) Model experimentation using [Azure ML](https://azure.microsoft.com/en-gb/overview/machine-learning/), again, using general computation targets such as [DSVM](https://azure.microsoft.com/en-gb/services/virtual-machines/data-science-virtual-machines/), [BatchAI](https://azure.microsoft.com/en-gb/services/batch-ai/), [Databricks](https://azure.microsoft.com/en-gb/services/databricks/) or [HDInsight](https://azure.microsoft.com/en-gb/services/hdinsight/).\n",
    "\n",
    "4) Model retraining using new data and a model obtained from the [Model Management](https://docs.microsoft.com/en-gb/azure/machine-learning/desktop-workbench/model-management-overview).\n",
    "\n",
    "5) Operationalization layer with a [Kubernetes cluster](https://azure.microsoft.com/en-gb/services/container-service/kubernetes/), which takes the best model and put it in production.\n",
    "\n",
    "6) Reporting layer to show the results.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/miguelgfierro/sciblog_support/master/Intro_to_Fraud_Detection/templates/fraud_detection_reference_architecture.svg?sanitize=true\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
